Skip to content

Add a per-language leakage dashboard renderer over benchmark runs#1757

Merged
maziyarpanahi merged 2 commits into
masterfrom
feature/om-575-per-language-leakage-dashboard
Jul 23, 2026
Merged

Add a per-language leakage dashboard renderer over benchmark runs#1757
maziyarpanahi merged 2 commits into
masterfrom
feature/om-575-per-language-leakage-dashboard

Conversation

@maziyarpanahi

Copy link
Copy Markdown
Owner

Description

Adds a per-language leakage dashboard over one or more benchmark runs so multilingual leakage trends, residual-PHI categories, and the worst labels are visible without exposing source text or report metadata. The dashboard produces both a self-contained sortable HTML report and deterministic companion JSON suitable for CI threshold checks.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature which changes existing behavior)
  • Documentation update
  • Code refactoring
  • Performance improvement
  • Test addition/improvement

Changes Made

  • Aggregate leakage counts and trends for every wired PII language from benchmark reports or OM-403 heatmaps.
  • Render one privacy-safe panel per language plus a sortable summary table, residual category breakdowns, and deterministic worst-label rankings.
  • Emit stable JSON with per-language evidence, optional leakage thresholds, combined gate status, and violations.

Testing

  • Added tests that prove the feature works.
  • New and existing unit tests pass locally.
  • Tested synthetic multilingual reports, direct heatmaps, single-language metrics, threshold failures, invalid inputs, and PHI exclusion.

Commands run:

  • make format
  • make lint
  • make format-check
  • make type-check
  • .venv/bin/python -m pytest tests/ -q — 5,205 passed, 46 skipped

Documentation

  • Added Google-style docstrings to the new public functions and classes.
  • Updated the changelog (not required for this scoped feature PR).

Code Quality

  • Ran make format, make lint, and make format-check.
  • Performed a self-review of the complete diff.
  • Added no new warnings or dependencies.

Dependencies

  • No new dependencies.

Checklist

  • Read the contributing and repository guidelines.
  • Commit has a clear, descriptive message.
  • Branch is based on current origin/master.

Related Issues

Closes #932

Depends on #650, completed by #680.

Screenshots/Examples

The generated HTML is self-contained and includes a sortable language summary followed by detailed language panels. Tests verify that all wired languages are present and that source text, model names, fixture identifiers, and metadata cannot enter either artifact.

@maziyarpanahi maziyarpanahi added roadmap-v2 OpenMed V2 roadmap backlog feature New capability P2 Medium labels Jul 19, 2026
@maziyarpanahi
maziyarpanahi force-pushed the feature/om-575-per-language-leakage-dashboard branch from 6514a7c to 6578ec5 Compare July 23, 2026 16:53
@maziyarpanahi
maziyarpanahi merged commit c1129ee into master Jul 23, 2026
35 of 39 checks passed
@maziyarpanahi
maziyarpanahi deleted the feature/om-575-per-language-leakage-dashboard branch July 23, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New capability P2 Medium roadmap-v2 OpenMed V2 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a per-language leakage dashboard renderer over benchmark runs

1 participant